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Executive summary 


Introduction 


This rapid evidence assessment aimed to investigate methods that schools could use to support remote learning during 
school closures caused by the 2020 coronavirus pandemic (Covid-19). The review sought to find the best evidence 
behind the wide array of approaches that schools might choose to use during the crisis. 


The review systematically searched for/through existing meta-analyses and systematic reviews to find the best evidence 
for a wide range of approaches. In addition to the direct teaching of pupils online by teachers, the review also found 
evidence for approaches that pupils can use independently, including with technology that adapts the content of tasks 
in order to provide additional challenge or support, educational games, and for approaches that support peer interaction. 


The review also examined blended learning approaches, which combine remote and classroom teaching. While these 
approaches may not be directly applicable during school closures, they were included in case there were useful lessons 
to be learned that were relevant to the current context of home learning. 


This report summarises the findings from 60 systematic reviews and meta-analyses answers under five key topic areas: 


- General remote teaching and learning 

- Blended learning 

- Computer-supported collaborative learning 
- Computer assisted instruction 

- Educational games 


Another crucial aspect of supporting pupil’s learning is parental engagement. Parental engagement approaches are 
beyond the scope of this review, but an existing EEF guidance report, Working with Parents to Support Children’s 
Learning (2019), and accompanying evidence review summarise the best available evidence for engaging parents in 
their children’s education. 


In each section, we have detailed the included reviews and listed Findings and implications drawn from the evidence. 
We have also highlighted case studies from projects funded by the Education Endowment Foundation in which similar 
approaches have been rigorously evaluated in English schools. 


Limitations 


It is important to note the limitations of applying existing evidence to school shutdowns due to Covid-19. None of the 
studies examined measure impacts on learning during a global pandemic, and there are characteristics of the current 
situation that will be unique, including due the demands of parents working from home, staff illness and other restrictions 
related to social distancing. 


In addition, there are other important limitations to the evidence. Few high-quality studies have looked at remote learning 
in school-aged education, and many of the included reviews combine evidence from school-aged education, university 
education and adult learners. This means that some approaches may be particularly applicable to older pupils, and 
careful consideration is required in assessing transferability to school contexts. Other evidence comes from 
programmes that have the potential to be delivered remotely — for example, intelligent tutoring systems — but that have 
to date largely been evaluated within classrooms. 


The rapid nature of the review also means that this review does not provide statistical analysis or comparisons between 
different approaches or reviews, but instead provides a narrative summary of findings. While each systematic review 
and meta-analysis included was appraised for quality, it was rare for the reviews themselves to appraise the quality of 
included studies comprehensively. This is a notable limitation of this body of evidence. 


These limitations notwithstanding, this report aims to summarise evidence that may support teachers in making 
decisions related to remote instruction during periods of school shutdown. 


Safeguarding 


When considering the remote teaching approaches discussed in this review, it is important to consider aspects of 
safeguarding. The National Society for the Prevention of Cruelty to Children (NSPCC) has produced resources on 
safeguarding during remote teaching, which can be found here. 


Key findings and implications 
1. Teaching quality is more important than how lessons are delivered 
Pupils can learn through remote teaching. 
Ensuring the elements of effective teaching are present — for example clear explanations, scaffolding and feedback — 
is more important than how or when they are provided. There was no clear difference between teaching in real time 


(“synchronous teaching”) and alternatives (“asynchronous teaching”). 


For example, teachers might explain a new idea live or in a pre-recorded video. But what matters most is whether the 
explanation builds clearly on pupils’ prior learning or how pupils’ understanding is subsequently assessed. 


2. Ensuring access to technology is key, particularly for disadvantaged pupils 
Almost all remote learning uses digital technology, typically requiring access to both computers and the internet. 


Many reviews identify lack of technology as a barrier to successful remote instruction. It is important that support is 
provided to ensure that disadvantaged pupils — who are more likely to face these barriers — have access to technology. 


In addition to providing access to technology, ensuring that teachers and pupils are provided with support and guidance 
to use specific platforms is essential, particularly if new forms of technology are being implemented. 


3. Peer interactions can provide motivation and improve learning outcomes 


Multiple reviews highlight the importance of peer interaction during remote learning, as a way to motivate pupils and 
improve outcomes. 


Across the studies reviewed, a range of strategies to support peer interaction were explored, including peer marking 
and feedback, sharing models of good work, and opportunities for live discussions of content. 


The value of collaborative approaches was emphasised in many reviews, although notably many studies involved older 
learners. Different approaches to peer interaction are likely to be better suited to different age groups. 


4. Supporting pupils to work independently can improve learning outcomes 


Pupils learning at home will often need to work independently. Multiple reviews identify the value of strategies that help 
pupils work independently with success. 


For example, prompting pupils to reflect on their work or to consider the strategies they will use if they get stuck have 
been highlighted as valuable. 


Wider evidence related to metacognition and self-regulation suggests that disadvantaged pupils are likely to particularly 
benefit from explicit support to help them work independently, for example, by providing checklists or daily plans. 


5. Different approaches to remote learning suit different tasks and types of content 


Approaches to remote learning vary widely and have different strengths and weaknesses. Teachers should be 
supported to consider which approaches are best suited to the content they are teaching and the age of their pupils. 


For example, games for learning were found to have a high impact on vocabulary learning in foreign languages, but 
there is less evidence related to their use in other subjects. 


Likewise, using technology to support retrieval practice and self-quizzing can help pupils retain key ideas and 
knowledge, but is not a replacement for other forms of assessment. 


The evidence in this review is drawn from diverse contexts that do not closely parallel the circumstances 
facing schools responding to Covid-19 in 2020. Many forms of digital technology could in theory be used to 
support remote learning, but are typically used in schools and have not been evaluated as remote learning 


tools. In all cases, it is important for teachers and school leaders to use their professional judgement in 
determining the support they provide their pupils and to monitor its impact on learning. 


Methods 


We undertook a rapid evidence assessment of existing systematic reviews and meta-analyses. We were guided by the 
Cochrane Collaboration Rapid Reviews Methods Group interim guidance on producing rapid reviews (Garrity et al. 
2020), as well as the Civil Service Rapid Evidence Assessment methodological guidance (Government Social Research 
Service, 2009) and the Cochrane Collaboration’s guidance on overviews of reviews (Pollock et al. 2020). A protocol for 
this Rapid Evidence Assessment, including more detail about the methodological approach taken, was published on 
the EEF website and is available here. 


Search criteria overview 


The scope was limited to systematic reviews and meta-analyses in order to respond quickly to the policy challenge of 
school closures due to Covid-19. The review was limited to studies from 2005 or later, as technology-based approaches 
from pre-2005 are unlikely to be relevant to current online learning approaches. To be included in the REA, a study had 
to meet the following additional criteria: 


e Population: include studies of interventions with primary or secondary aged pupils. We included reviews of 
studies from any country. 

e Intervention: include studies of distance learning approaches as methods of teaching that take place entirely 
outside of the classroom environment. They could be synchronous or asynchronous. The primary objective 
needed to be pupil learning outcomes. Examples of included approaches could be: 

o Recreating the classroom environment through online platforms such as “Google Classrooms” or 
equivalents 

Online tutoring that provides intensive support to individual pupils through a two-way link 

Digital platforms/education software that are used independently by pupils 

Online lectures without pupil interaction 

Games for learning (GL) 

We included blended learning approaches but examined them separately for information on best 

practice on schools setting learning for the home environment. 

e Comparison: compare distance learning to classroom or school-based learning or other types of distance / 
online based approaches 

e Outcome: cover any measure of academic achievement or cognitive measure of ability of pupils in any 
subject. 

e Study design: be a meta-analysis or systematic review of distance learning effectiveness or implementation of 
distance learning 


OOO? .O; <0 


How the review was conducted 
We undertook the following steps to produce the REA, as described in the protocol: 


1. Searched for relevant studies, including searching bibliographic databases and known sources of systematic 
reviews. 

2. Screened the search results for inclusion using the criteria described above using a two-stage process, first 
screening at title and abstract and then at full text level. 

3. Extracted data from each included systematic review and meta-analysis, including information on methodological 
and substantive features, results, specifically average effect sizes and associated confidence intervals where 
presented, and any information around barriers and facilitators to successful implementation of distance learning 
approaches. 

4. Appraised each systematic review and meta-analysis for methodological quality, to make a judgement on how 
much confidence to place in the findings of each review. 

5. Summarised the findings of each included review, grouped by the broad topic they address. 


Figure 1: PRISMA diagram of search process results 
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The initial searches returned 3846 references. After removing duplicates and screening on title and abstract, 244 
references were labelled for inclusion. We were unable to retrieve 4 references, which meant that 240 were screened 
on full text. Of these 240, 60 reviews were included in the review. All 60 had data extracted. After data was extracted, 
findings fell into five broad categories, which are summarised in sections below: 


- General remote teaching and learning (10 reviews) 

- Blended learning (5 reviews) 

- Computer assisted collaborative learning (5 reviews) 
- Computer aided instruction (29 reviews) 

- Learning games (14 reviews) 


Some reviews covered multiple categories and are included in multiple sections. Where different pooled effects are 
reported in the review, these are reported in the relevant section. 


Remote teaching and learning 


The intervention 


This section examines the reviews that have examined remote learning generally — it includes studies that have tried to 
measure the impact of combined approaches to distance learning and teaching directly through video conferencing 


software or other methods of online tuition. 


Findings and implications: 


e Whether the elements of effective teaching are present is less important than how or when they are provided. 

e There is some evidence that encouraging peer interactions can lead to improved learning outcomes, 
particularly for older pupils, and that prompting pupils to reflect on their own learning during remote learning 
may be an effective approach. 

e Overall, evidence about remote learning is limited and draws on studies from higher education, as well as on 


and older reviews. Findings are likely to be less applicable to primary-aged children than older pupils and may 


not reflect the latest technological changes. 


Relevant studies 


Review 


Review focus 


Number of studies 


Reported impact 


Bernard et al 


An effect is provided for general distance learning. The review examines 
the impacts of different types of interaction in distance learning (for 


74 


0.38 


Chauhan (2017) 


generally. There is a specific focus on e-learning and the extracted 
effect size is for learning in informal settings, and not the overall impact 


education settings, 


e009) example, student-student or student-teacher) 
An effect is provided for general distance learning. The review then 

eet Oye al examines the differences in types of student-student interaction in 36 0.31 
distance learning. 
The meta-analysis covers technology impacts on education more 21 (for non-formal 0.70 


(non-formal education 


learning approaches. 


of technology. e.g. home) settings, e.g. home) 
Darah Seal This meta-analysis examined the impact of planned discussions as part 
(2013) : of online teaching. The reported effect compares the impact of strategic 8 0.50 
discussion compared with conventional discussion in online learning. 
Jopling (2012) This paper reviews the literature for online one-to-one tuition. 17 N/A 
Méaneotal This review combined a meta-analysis of distance learning and blended 
(2009) , learning approaches with a qualitative review of remote learning 46 0.24t 
approaches that took place exclusively with school-aged pupils. 
Means et al. This review updates Means et al. (2009) removing coding errors in the 45 veo 7 
(2013) earlier analysis and providing additional moderator analysis. (0.14 p<.05 for online 
only instruction). 
Sandy-Hanson This review looked generally at the use of technology in supporting 5 (remote 0.26 
(2006) pupils learning, but contained separate analysis of remote learning. learning) . 
: This review examined the studies located by Means et al (2009) to 
a ull; Straub identify the impacts of remote learning approaches for pupils with 4 0.38 
special educational needs (SEN). 
hae Vaal This meta-analysis examines distance learning in comparison to face-to- 
(2005) : face education. It combines solely online approaches and blended 51 0.10 


' This effect size had been updated in Means et al 2013 to remove coding errors. The effect size is reported here for record but is superseded by 
the later study. 


How secure is the evidence? 


A number of meta-analyses have examined the impact of remote or distance learning. The evidence base is limited by 
the small number of rigorous studies that have taken place within school-aged education. Means et al (2009) reviewed 
distance learning approaches for children from kindergarten to grade 12 (4 to 19 year olds) in the USA and only located 
9 studies that met the inclusion criteria. In order to estimate the impact of remote learning, they combined studies of 
school aged pupils with studies from higher education. While analysis within several of the reviews has not shown a 
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significant difference in outcomes between higher and compulsory education, the fact that many of the findings on 
successful types of remote learning are drawn from contexts other than school is a severe limitation to the applicability 
of the evidence. 


We also appraised the quality of each included systematic review and meta-analysis. It was rare for the included reviews 
to appraise the quality or risk of bias of their included studies, a limitation of this body of evidence. We therefore do not 
have aclear understanding of the quality of the underlying research contributing to our conclusions. 


What does the evidence say? 


The limited amount of rigorous evidence notwithstanding, reviews that have examined the impact of distance learning 
have found that results are equivalent or slightly higher for pupils who have been taught through distance instruction 
rather than traditional face to face instruction. One study found that results are higher when blended and online learning 
are analysed together (Means et al 2013); when online teaching was examined individually the impact was found to be 
no different from traditional instruction. 


One review (Bernard et al 2009) examined different types of interaction within distance learning, comparing student- 
student, student-teacher, and student-content, interaction. The review found evidence that student-student interaction 
had the largest impact on student outcomes. This finding is corroborated by the review by Means et al. (2013), which 
found a positive impact for interaction with peers, whether synchronous or asynchronous. A follow-up review to Bernard 
et al (Borokhvski et al 2012) found that when student-student interaction was designed into the distance learning the 
impact was even greater. These findings should be treated with caution due to the fact that, despite the review searching 
for school-aged interventions, the evidence was drawn exclusively from higher education studies. When another review 
(Darabi et al. 2013) examined the impact of planned strategic discussions between peers in comparison to conventional 
online discussions, the impact for all pupils was positive but there was no impact for school-aged pupils when higher 
education studies were removed from the analysis. 


Several reviews explored different ways of presenting information and teaching. Means et al (2013) found no evidence 
that adding multimedia to online instruction had any impact on learning outcomes, concluding that the teaching was 
more important than the medium itself. There was, however, some evidence that supporting learners to reflect on their 
own learning had positive impacts. Zhao et al. (2005) found evidence that a combination of asynchronous and 
synchronous learning was most beneficial to outcomes. This finding is not replicated across other reviews, which found 
no evidence to differentiate either asynchronous or synchronous instruction (Means et al 2013 and Bernard et al 2009). 


The evidence on remote learning for pupils in primary school is very limited. Future research is needed on the efficacy 
of remote learning for younger pupils. 


How does the approach impact disadvantaged pupils? 


None of the studies examined the impact of distance learning techniques on disadvantaged pupils. One study that 
focused on the implementation of one to one tuition highlighted the importance of access to technological infrastructure 
(Jopling 2012). 


Relevant EEF-funded evaluations 
Affordable Maths Tuition 
The EEF has not funded any evaluations of school-wide distance learning 
approaches. There may be some transferable learning from one project, 
‘Affordable Maths Tuition’, in which pupils received maths tuition from trained Testing the impact of online maths tutoring 
maths graduates based in Sri Lanka and India. 


Teachers were largely positive about the online tuition, and reported that it RoE Poncent Evalue 
appeared to improve pupils’ comprehension, verbal fluency, and confidence in The York Trials Unit 
maths. However, the impact evaluation found no evidence that the intervention 

had an impact on Key Stage 2 maths, compared with ‘business as usual’ Schools jrant 
teaching and support in Year 6. einaaos 
Some pupils complained they were interrupted when working as the tutor could 

not see that they were still tackling a problem. This may have been due to the 

lack of face-to-face contact with the tutors. 


O Organising your school 


Tuition did not take place in the home learning environment, so findings may not 
be transferable, it is important schools consider how to track pupils progress with 
tasks and remove technological barriers to receiving instruction. The full 
evaluation report of ‘Affordable Maths Tuition’ can be found here. 


Blended learning 


The intervention 


Blended learning combines aspects of online instruction with classroom teaching or activities. This means that the 
evidence may not be directly applicable to schools during periods of shutdown. These studies have been included to 
give insight into successful implementation of the distance components of blended learning approaches. 


Findings and implications 


e Lack of access to technology, particularly for disadvantaged pupils, is identified as a key barrier to the 
success of blended learning approaches. 

e Peer marking was identified as a promising component of some blended learning schemes — particularly 
when pupil work and feedback is anonymised. 

e Providing professional development to teachers implementing new approaches and using new platforms is 
identified as important. 


Relevant studies 


Review Review focus Number of studies Reported impact 
Cui and Zheng This review focuses specifically on using peer evaluation - 
(2018) typically sharing essays with peers for feedback - in blended 23 0.68 
learning environments. 
This review combined a meta-analysis of distance learning and 
Means et al. blended learning approaches with a qualitative review of remote 46 0.24 
(2009) learning approaches that took place exclusively with school-aged : 
pupils. 
: : F . ; 0.20 
Means et al. This review updates Means et al. (2009) removing coding errors in 45 0.14 0<.05f F | 
(2013) the earlier analysis and providing additional moderator analysis. (0. phibe Seana 
instruction). 
Poirier et al. This review provides a narrative summary of different blended 14 N/A 
(2019) learning approaches. 
Rasheed et al. This review summarises barriers to implementing blended learning 
: 30 N/A 
(2020) approaches effectively. 


How secure is the evidence? 


Only one review that measured the impact of blended learning approaches on pupil outcomes was identified (Means et 
al 2013). This study was conducted seven years ago and included a large number of studies from higher education. 
Two of the other reviews provided a narrative synthesis of blended learning approaches and important aspects of 
implementation, while the final review (Cui and Zheng 2018) focused specifically on the use of peer evaluation in 
blended learning environments. While some insights may be gained from the identified studies — in particular through 
identifying potential barriers to implementing blended learning approaches — practitioners should be cautious when 
applying findings to their own context. 


We also appraised the quality of each included systematic review and meta-analysis. It was rare for the included reviews 
to appraise the quality or risk of bias of the included studies, a limitation of this body of evidence. We therefore do not 
have aclear understanding of the quality of the underlying research contributing to our conclusions. 


What does the evidence say? 


The one review that measured impact of blended learning approaches on learning found a positive result (Means et al 
2013). Poirier et al (2019) conducted a systematic review into the evaluation of blended learning approaches and found 
that the evidence was mixed. 


Similarly to the general reviews of remote learning, the value of communication between pupils was a highlighted by a 
number of the reviews (Means et al 2013, Poirer et al 2019, Cui and Zheng 2018). Poirer et al noted that a feature of 
some of the positive evaluations was the facilitation of peer to peer communication, examples including message 
platforms or online forums. Given the small number of studies, this result may not be transferable. Cui and Zheng (2018) 
examined peer evaluation more systematically across blended learning environments and found a positive effect. One 
of the interesting findings of the Cui and Zheng analysis is that anonymous peer marking of work had a higher impact 
on pupil outcomes, and that peer reviewing that was supported by teachers had a higher impact. Peer tutoring is a well- 


evidenced approach in traditional teaching settings — for more information, see the Education Endowment Foundation 
Teaching and Learning Toolkit entry on Peer Tutoring, which can be found here. 


Rasheed et al (2020) conducted a systematic review of implementation challenges of the online component of blended 
learning. Some of the highlighted challenges included supporting teachers’ professional development in implementing 
online aspects of blended learning and supporting the digital literacy of pupils. A key challenge identified was supporting 
the self-regulated learning of pupils during online teaching and preventing feelings of isolation from pupils. Technological 
challenges were also identified as a barrier to implementing online components of blended learning. 


How does the approach impact pupils from disadvantaged backgrounds? 


While none of the reviews directly explored the impact of blended learning on disadvantaged pupils, many of the 
challenges identified in the Rasheed et al review might be considered particular barriers to the learning of disadvantaged 
pupils. Ensuring access to technology and reliable internet access is crucial to blended learning approaches. 


Relevant EEF-funded evaluations 


The EEF funded an evaluation of a project called ‘Flipped Learning’, in which Flipped Learning 
pupils learned core content online and then used class time for activities to re- 
enforce learning. 


Pupils undertake some learning prior to lessons, freeing 


In the study, pupils accessed a learning platform where they could access Up Glass lime: for feedback and personalised support 
resources uploaded by teachers communicate with other members of the class. 

Teachers also had access to a resource bank based on the learning objectives Independent Evaluato 

for mathematics according to the national curriculum IEE 


The Institute for Effective Education 


The evaluation found a small positive impact on Key Stage 2 maths outcomes. 
The outcome was larger for pupils eligible for free school meals. 


An important finding from the study was that some pupils were unable to Themes 

engage with the online component of the project due to not having access to 
the internet at home. In this project, pupils were given access to internet through 

homework clubs that took place in school. In situations where access to school 

is not possible, practitioners need to think carefully about how to either provide 
access to the internet to all pupils or to provide teaching through offline means. 


The full evaluation report of ‘Flipped Learning’ can be found here. 
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Computer-supported collaborative learning 


The approach 


In computer-supported collaborative learning (CSCL) approaches, learning takes place through social interaction using 
either a computer or the internet. Collaboration may occur using video, chat, emails, discussion boards or knowledge 
forums. These approaches can take place in the classroom or over the internet. Approaches often use specific tools for 
co-construction of work — for example, shared workspaces like Google Docs or Slack. Approaches in which pupils 
interact over the internet while working at home might be relevant during school closures. 


Findings and implications 


e Providing elements of collaborative learning within online learning environments appears to be linked to positive 
impacts on learning outcomes. 

e The impact of computer-supported collaborative learning has mainly been reviewed in STEM subjects and 
language learning. 

e There is some evidence that shared online workspaces like Google Docs can support learning, particularly for 
older pupils. However, it is likely that pupils will require on-going support to use them effectively. 


Relevant studies 


Review Review focus Number of studies Reported impact 


This review examines the impact of collaborative 
Chen et al (2018) learning in computer environments. The headline impact 84 0.420 
compares CSCL with independent computer use. 


This review is an extension of Chen et al (2018) which 
Chen et al. (2019) | adds extra moderator analysis. All other aspects of the 84 N/A 
review are the same. 


This review examines the impact of CSCL in STEM 


sabods 143 0.490 


Jeong et al. (2019) 


This review examines CSCL for second language 
Lin Huifen;(2014) learning — primarily examining directly communicating in 59 0.441 
a second language online. 


This review does not compare CSCL with traditional 
teaching, but instead compares the impact of using 12 0.000 
argumentation or not within CSCL environments. 


Wecker and 
Fischer (2014) 


How secure is the evidence? 


There are several recent meta-analyses looking at the impact of computer supported collaborative learning. The key 
challenge of this evidence base is that it is not always made clear when computer-supported collaborative learning 
(CSCL) interventions are taking place online or in school environments. 


Another challenge when interpreting these reviews is that several of them examine specific subjects, interventions or 
comparisons, so caution should be applied when transferring these findings to other contexts. For example, Lin (2014) 
examines the impact of online communication in second language learning. The oral focus of second language 
outcomes may mean that online communication is particularly suited to language learning in a way that does not easily 
transfer to other subjects. Wecker and Fischer (2014) examine the impact of argumentation within CSCL environments. 
This means that the reported effect size is not a measure of CSCL overall, as comparison pupils were receiving CSCL 
without elements of argumentation. 


The most general overview of CSCL is provided by Chen et al. (2018). This study is not limited to school-aged pupils, 
and while moderator analysis found no difference in outcome between school-aged pupils and older learners, it is 
important to remember that analysis of different CSCL approaches within the review will contain learners from different 
populations. The main comparison reported by Chen et al (2018) is between collaborative learning in computer 
environments and independent learning in computer environments and so should not be interpreted as the impact of 
CSCL in comparison to traditional teaching. 


We also appraised the quality of each included systematic review and meta-analysis. It was rare for the included reviews 
to appraise the quality or risk of bias of the included studies, a limitation of this body of evidence. We therefore do not 
have aclear understanding of the quality of the underlying research contributing to our conclusions. 


What does the evidence say? 


The evidence for computer-supported collaborative learning (CSCL) approaches is consistently positive across the 
included reviews. Even when just examining school aged pupils Chen et al (2019) found significant positive results 
when comparing collaborative learning with independent learning using computer-based instruction. One study reported 
a specific outcome measure for using CSCL as part of distance learning and found that the impacts were positive and 
of a similar magnitude to overall CSCL outcomes (Jeong et al 2019). Chen et al (2018) also explored the impacts of 
computer use for collaborative learning and found positive impacts on social interaction and group task performance. 


An interesting finding from the reviews is the way CSCL appears to have been used in specific subjects. Chen et al 
(2019) found that at primary school all of the studies focused on using CSCL for science, maths or language instruction. 
At secondary level all studies were focused on maths or science. Jeong et al (2019) found that CSCL had positive 
impacts for STEM subjects, while Huifen (2014) found a positive impact for second language learning. 


When studies examined the way peers communicate they found no difference between synchronous or asynchronous 
communication (Chen et al 2018 and Lin 2014). There were, however, some insights into successful implementation 
strategies. Chen et al (2018) found that peer assessment and feedback strategies led to positive learning outcomes. 
They also noted that providing instruction and guidance within the computer platform led to positive outcomes. Jeong 
et al (2019) explored different types of implementation and their impact on STEM outcomes. The combination of 
technology was found to be particularly important. For example, video conferencing had a large positive impact when 
paired with shared online workspaces, for example, Google docs. When video conferencing was only supported by 
email communication, there was no evidence of impact. 


How does the approach impact pupils from disadvantaged backgrounds? 


None of the reviews considered the impact of computer-supported collaborative learning (CSCL) approaches on pupils 
from disadvantaged backgrounds. Future research is required to explore how disadvantage pupils are impacted by 
CSCL. Schools should carefully monitor the learning outcomes of disadvantaged pupils and remove any barriers that 
pupils may have to accessing CSCL approaches — for example, lack of required technology. 


Computer-assisted instruction 


The approach 


Computer-assisted instruction (CAI) covers a range of approaches in which pupils receive instruction through digital 
technology. Some common approaches include computer-based interventions that scaffold practice and intelligent 
tutoring systems that often give adaptive feedback to learners as they work. Many of these interventions have largely 
been used within school environments, and many of the reviews covered below summarise results from evaluations 
that have taken place with teacher support present. In response to Covid-19 many companies are offering digital 
learning applications or programmes in which pupils learn independently. For this reason, we have chosen to include 
reviews of CAI approaches in our summary. 


Findings and implications 


e Many computer-assisted instruction programmes have only been evaluated in school settings with teacher 
support present and may not be transferable to remote learning. 
e The impact of computer-assisted instructions vary widely. However, several reviews identify scaffolding and 
feedback as components of successful approaches. 
e Ensuring that teachers and pupils are provided with support and guidance to use technology is essential, for 
example if new approaches such as intelligent tutoring systems are being introduced. 


Relevant studies 


(2019) 


environments. 


Review Review focus SUE ee Sepals 
studies impact 

Abrami et al. (2019) He review examined the impact of ABRACADABRA — a balanced approach to early 17 0.26 
iteracy instruction. 

Batdi (2015) This meta-analysis measured the impact of computer based teaching on learners 78 1.34 
academic success. 
This Bayesian meta-analysis measured the within subject impact of computer-based 

Belland et al. (2017) scaffolding approaches. The outcome described is for STEM subjects in primary 56 0.74 
school. 

Belland et at (2017b) This review measures the impact of computer-based scaffolding on STEM outcomes. 114 0.46 

Cheung and Slavin This review is a general review of the impact of digital technology. The effect size 74 0.16 

(2013) described is specific to computer assisted instruction. : 

pec p —___ : 

Chiu (2013) This meta-analysis summarises the impact of computer mediated instruction for 16 0.75 
language learning and teaching. 
This meta-analysis focuses specifically on providing feedback in computer-based 

der Kleij et al. (2015) environments. Multiple outcomes are described — the reported impact is for 40 0.39 
elaborated feedback vs knowledge of correct results. 

Fang et al (2019) This review measures the impact of the ALEKS intelligent tutoring system. 15 0.10 
This paper examined the impact of automated guidance on school-aged pupils’ 

Gerard et al. (2015) learning outcomes. 41 0.34 

Karich et al. (2014) ee eee measured the impact of learner control within computer assisted 29 0.05 

Kim et al (2018) This review examined the impact of computer-based scaffolding when applied to 21 0.39 
problem-based learning. 

eat Pitcher This meta-analysis measured the impact of intelligent tutoring systems. 50 0.62 

Kunkel (2015) This review measured the impact of instruction provided by an application rather than 13 0.14 
a teacher or classroom aide. 

Ma et al. (2014) This review measured the impact of intelligent tutoring systems. 107 0.41 
This review examined computer assisted instruction of maths for students with 

Serta ey) learning disabilities ey ia 

Rigney et al. (2020) This review examined the evidence for Headsprout - a computer assisted instruction 6 NA 
programme. 

. This review examined digital technology more generally, the impact summarised is 
Sandy-Hanson (2006) specific to computer-based instruction 19 (lor. CAl) ues 
Sharifi et al. (2018) ees measured the impact of computer assisted instruction on language 140 0.50 
? This review examines approaches to improve reading in primary and middle school 

lav BU Al NEDO) generally. The impact described is specific to computer assisted instruction. sara 

Steenbergen-Hu and , : : : ; : 

Cooper (2013) This meta-analysis measures the impact of intelligent tutoring systems. 26 0.09 

Strong et al. (2011) This meta-analysis summarises the impact of the Fast ForWord programme. -0.26 
This review examines the impact of computer supported scaffolding of reading 

ter Beek et al. (2018) comprehension N/A 

Thomas et al (2013) This meta-analysis compares the impact of interactive and didactic learning within 40 0.18 
computer assisted instruction. 

Verschaffel et al This review examines the use of metacognitive scaffolding in ICT based learning 29 N/A 
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Weng Pei-Lin et al. This meta-analysis measures the impact of computer-assisted instruction for SEND 8 0.35 
2014 upils. ; 
What Works : ' F Mita ae : : 

Clearinghouse (2009) Earobics - interactive software providing instruction in early literacy skills. 4 0.49 
What Works ' ‘ : 

Clearinghouse (2007) This report summarises the evidence for the Read Naturally programme. 2 N/a 
pote area This review examines the use of technology to teach writing skills. 29 N/A 
Xu et al. (2019) This meta-analysis measures the impact of intelligent tutoring systems. 19 0.60 


How secure is the evidence? 


One of the challenges of the interpreting the evidence for computer-assisted instruction is the large number of reviews, 
which focus on a variety of different approaches and have been conducted with different levels of methodological rigour. 
Many of the reviews of CAI approaches have been published within the last five years, so should capture recent 
developments in technology. 


Perhaps the most important limitation of the evidence for CAI approaches is that most of the evidence comes from 
classroom settings, in which teachers or classroom assistants are able to support learning. This means that caution 
should be applied when transferring these findings to distance learning contexts. 


Another important limitation of some of the included reviews is they include outcome measures that are very close to 
the intervention being delivered. This issue is particularly prominent in the examples where CAI approaches have been 
used to teach vocabulary in second language learning and can lead to very high effect sizes being reported. 


We also appraised the quality of each included systematic review and meta-analysis. It was rare for the included reviews 
to appraise the quality or risk of bias of the included studies, a limitation of this body of evidence. We therefore do not 
have aclear understanding of the quality of the underlying research contributing to our conclusions. 


What does the evidence say? 


The overall impact of computer-assisted instruction approaches varies. Reviews report outcomes that range from high 
(Batdi 2015, Chiu 2013, Kulik and Fletcher 2016) to low or negative (Steenbergen-Hu and Cooper 2013, Strong et al. 
2011). These differences may be caused by genuine variation in the efficacy of different CAl approaches as well as 
variation in the methodology of reviews that examine them. There is, however, clear evidence that CAI approaches 
have the potential to improve learner outcomes, with many of the reviews finding positive impacts. 


Intelligent tutoring systems (ITS) are a popular intervention delivering computer-assisted instruction. ITS aim to deliver 
personalised instruction or feedback to learners. Several reviews have specifically examined the impact of ITS. Ma et 
al. (2014) found that ITS are effective at improving learner outcomes, but that the impact is dependent on the 
comparison condition for the intervention. When compared to whole class teaching, intelligent tutoring systems were 
found to have a high impact, but when compared to one to one human instruction impacts were negative and non- 
significant. Xu et. al (2019) found positive results even when compared to human tutoring. Steenbergen-Hu and Cooper 
(2013) found a much smaller positive effect, concluding that intelligent tutoring systems seem to have comparable or 
slightly better outcomes than traditional classroom instruction. 


Several reviews identify scaffolding and feedback as important ingredients of successful CAI approaches. Scaffolding 
is identified as leading to positive student outcomes by Gerald et al. (2015), Belland et al. (2015), Belland et al. (2015b) 
and Kim et al. (2018). Verschaffel et. al (2019) find several studies indicating that specifically metacognitive scaffolding 
— for example, prompts either from teachers or built into the technology that encourage learners to think about 
successful strategies for learning or when to request help — is an important component of successful CAI. Kim et al 
(2018) also found higher impact from metacognitive and strategic scaffolding than conceptual scaffolding. The use of 
feedback within CAI corroborates the wider evidence on feedback, with elaborated feedback in CAI leading to higher 
outcomes than simply informing pupils of their results (der Kleij et al. 2015). 


Implementation seems particularly important across CAI approaches. Kulik and Fletcher (2013) studied the comparison 
between strongly and weakly implemented examples of intelligent tutoring systems and found that examples with weak 
implementation had no positive impact. A characteristic of successful ITS approaches identified in the study was that 
teachers used ITS to supplement teaching rather than as a direct replacement. If schools decide to introduce new 
technology during periods of closure, they should consider how to make sure approaches are implemented successfully 
and supplement existing learning approaches. 


Several of the reviews summarise the evidence for specific branded interventions that can be delivered using 
computers. For example, two What Works Clearinghouse reviews examine the Earobics programme, which had some 
evidence of promise (What Works Clearinghouse 2009) and Read Naturally (What Works Clearinghouse 2007), which 
was found to not have enough high quality evidence of impact. Other named interventions include ALEKS — an intelligent 
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tutoring system that was found to be comparable to usual class teaching (Fang et al. 2020); Headsprout — a computer 
based reading approach that was found be effective at improving reading comprehension (Rigney et al. 2020); and Fast 
ForWord an computer-based language intervention that was not found to be effective (Strong et al. 2011). One of the 
named interventions with the highest impact was ABRACADABRA, a balanced reading approach that can be delivered 
digitally (Abrami et al. 2019). While all of these interventions can be delivered using computers, it is important to 
emphasise that these reviews and their results do not reflect the programmes being implemented in the home learning 
environment and several explicitly use teacher or teaching assistants to support the delivery in current studies. 


How does the approach impact pupils from disadvantaged backgrounds? 


Belland et al. (2017) found a positive impact for computer-based scaffolding approaches for pupils from low-income 
backgrounds. This contrasts with Cheung and Slavin who found higher outcomes for schools with high SES, although 
they acknowledge the possibility that this is a result of sampling error and not a generalisable finding. Kunkel (2015) 
noted that many of the schools included in studies of CAI had high proportions of pupils from disadvantaged 
backgrounds. 


While the evidence for the impact of CAI on disadvantaged pupils is limited and has mixed outcomes, it is worth nothing 
that technological access is an especially important barrier to programmes that rely on adaptive feedback being 
delivered through apps or computers. If this barrier can be overcome, CAI does have the advantage of being able to be 
targeted at particular groups of pupils that might benefit from additional support. 


Relevant EEF-funded evaluations 


The EEF funded an evaluation of onebillion — a maths learning app that is used by 
pupils in Key Stage 1 to develop early mathematical skills. 


In the EEF trial, the app was used as a targeted intervention for small groups of 
pupils identified as being in the lower half of the class for maths performance. Pupils 
used the app independently on tablets, while being monitored by teaching assistants. 


The independent evaluation found that onebillion had a positive impact on maths 
learning overall. There was, however, a negative impact on pupils eligible for free 
school meals — although the number of pupils was not large enough for this finding 
to be generalisable to all pupils eligible for free school meals. 


It is important to highlight that in the EEF trial the intervention took place within 
school, with teaching assistants monitoring pupils to make sure they were using the 
app. 


The full evaluation report for ‘onebillion’ can be found here. 


Another evaluation funded by the EEF tested the impact of Abracadabra (ABRA), 
one of the specific CAI interventions identified in the reviews. 


ABRA is an online toolkit of phonics, fluency and comprehension activities aimed at 
primary school pupils. In the EEF trial, the use of ABRA in school settings was 
supported by trained teaching assistants. The independent evaluation found a 
positive impact on Key Stage 1 reading outcomes. Schools that want to use ABRA 
as a tool for distance learning should consider how they can provide the support 
pupils need to use the online tools effectively. 


The full evaluation report for ABRA can be found here. 
The EEF has also published independent evaluations of other CAI approaches: 


e Accelerated Reader 
e = Tutoring with Alphie 
e Stop and Think: Counterintuitive Concepts 


None of the evaluations took place during distance learning. 


onebillion: app-based maths learning 


University of Notting! 


%* promising project 


Maths learning app monitored by teaching assistants 


1dependent Evaluato: 


University of Oxford 


Abracadabra (ABRA) 


entr 
( % promising project ) 


An online reading programme to improve early literacy 


ndependent £ 


London School of Economics 


© Organising your school 
L__ Language and literacy 


Games for learning 


The approach 


Games for learning (GL) are often described as ‘serious games’ and are designed for educational purposes. They are 
often played on computers with learning taking place individually. They are contrasted with entertainment games, in 
which the aim is to provide entertainment rather than improving learning outcomes. While many evaluations of digital 
games have taken place in classroom settings, the fact that they can be delivered without face to face instruction 
means that schools may consider using them as part of distance learning provision. 


Findings and implications 


e Many games for learning have only been evaluated in school settings and may not be well-suited to supporting 
remote learning. 

e There is some evidence that the games that are well-suited to remote learning have particular characteristics, 
such as providing additional scaffolding or detailed feedback. 

e The evidence base is stronger for using games to learn foreign languages. Some studies also highlighted that 
games involving peer interaction were particularly promising. 


Relevant studies 


Review Review focus Number of Reported impact 
studies 
Acquah and Katz (2020) Enaece eerie digital based games learning and L2 26 N/A 
Boyle et al (2016) ening GEE review examines game-based approaches to 143 N/A 
This meta-analysis measures the impact of digital game based 
Byun and Joung (2018) learning in mathematics. 33 0.370 
This meta-analysis measures the impact of digital game based 
Chen et al. (2018) learning of vocabulary. 10 1.027 
Girard et al. (2013) This review summarises the evidence for serious games. 11 N/A 
Hainey et al. (2016) This review summarises the evidence for game based learning. 45 N/A 
; Thi i he | f : based | ; 
; is review summarises the literature of game-based learning on 
Husealpetals (2019) science outcomes for primary school pupils. w ne 
fen This review summarises the literature of game-based learning on 
Martinez-Garza et al. (2013) Sclonee OUIGORIGd 56 N/A 
‘ This meta-analysis summarises the impact of serious games on 
Riopel et al. (2019) natural sciences outcomes. 12 sai 
Tokac et al. (2019) This review measures the impact of game-based learning on 24 0.130 


mathematics outcomes in school-aged pupils. 
This meta-analysis measures the impact of digital vocabulary 


Tsai and Tsai (2018) learning games on vocabulary outcomes of foreign language 26 0.986 
students. 
This review measures the impact of serious games on motivation 
HHOUIeIS Chel (etty) and learning. The outcome summarised is for learning. ou i 
Wouters and van Oostendorp This meta-analysis compares the impact of games that feature 29 0.330 
(2013) instructional support with games that do not. : 
Zou et al. (2019); This review examines the evidence for digital game-based learning 21 N/A 


of vocabulary. 


How secure is the evidence? 


Many of the included reviews did not calculate the impact of games for learning, and several discuss the lack of high 
quality evaluations of the approach. As with computer-assisted instruction, many of the reviews either do not make clear 
what context the games have been evaluated in, or comprise largely of evaluations that take place within school, with 
support from teachers. This means that findings may not be transferable to remote learning. Also, similar to computer- 
assisted instruction approaches, several of the reviews focus on language learning and have vocabulary outcomes that 
may lead to larger effect sizes that do not translate to other subjects. 


We also appraised the quality of each included systematic review and meta-analysis. It was rare for the included reviews 
to appraise the quality or risk of bias of the included studies, a limitation of this body of evidence. We therefore do not 
have a clear understanding of the quality of the underlying research contributing to our conclusions. 
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What does the evidence say? 


Among the reviews that calculate an overall impact for games for learning, there are a wide range of effects. The two 
studies with the highest impacts (Tsai and Tsai 2018, Chen et al. 2018) both examine the impact of digital games on 
foreign language learning. GL may be particularly effective for learning and memorising vocabulary from foreign 
languages or that vocabulary outcome measures are more likely to produce high results in evaluations of GL 
approaches. Several of the reviews call for more high-quality evaluation of GL and note mixed results of evaluations 
that measure standardised learning outcomes (Hainey et al. 2016, Hussein et al. 2019). Despite these limitations, the 
seven meta-analyses of GL that calculate an average impact do all find the approach to be positive. It does appear that 
using games for learning has the potential to improve pupil outcomes. 


There are some indications that different types of learning games are applicable in different contexts. For example, Tsai 
and Tsai (2018) found that more sophisticated games had a high impact in informal settings, such as home learning, 
but did not have an impact when delivered in formal education. 


Wouters and van Oostendorp (2013) reviewed the impact of instructional support being provided within games and 
found a positive impact. Other aspects of games that were found to be linked with positive outcomes were aspects of 
reflection, modelling, feedback and personalisation. This finding is similar to positive characteristics identified in the 
reviews of more general computer-assisted instruction. Riopel et al (2019) also found that learner control over content 
sequence or pace led to positive impacts. 


Several of the reviews provided a narrative summary of different evaluations of GL approaches and discussed aspects 
that had been identified as characteristics of successful games. Boyle et al (2016) identified the value of games that 
provided rewards and also discussed the importance of usability — for example, games with fast loading times. Aquah 
and Katz (2020) highlighted the importance of games being easy to use, providing instant feedback and having a clearly 
defined goal. They also highlighted the benefits of allowing player interaction. An important limitation pointed out by the 
review is that games being engaging didn’t seem to correlate with positive learning outcomes — it is possible to make 
games that are entertaining but do not provide learning benefits. It is important to note that many of the insights from 
these narrative summaries are based on a very small number of studies and may not be generalisable. 


How does the approach impact pupils from disadvantaged backgrounds? 


No reviews systematically examine the impact on pupils from disadvantaged backgrounds. One review (Martinez-Garza 
et al. 2013) highlighted an example of a hand-held maths game that boosted the learning of disadvantaged first and 
second grade students in the USA. Another review highlighted a study in which encouraging competition between 
students had an impact on attainment gaps within the class — although this focused on the gap between high- and low- 
attaining pupils rather than gaps based on the socio-economic status of pupils. Hussein et al. (2019) identify 
technological capacity as a barrier to implemented GL approaches. 


Relevant EEF-funded evaluations 


The EEF has funded one evaluation of a game for learning approach, _ 
‘GraphoGame Rime’. This is a computer game designed to teach pupils to GraphoGame Rime 
read by developing their phonological awareness and phonics skills. Pupils University of Cambridgé 
access the game on individual devices. 
Testing a computer programme designed to improve 
The independent evaluation did not find a positive impact for pupils identified pupils’ literacy through teaching phonics via “rhyme 
as having low skills by the phonics screening test in Key Stage 1. Pupils in analogy 
comparison schools did receive additional support, including small group and 
one to one activities. Teachers found the programme easy to implement and 
considered it highly engaging. 


This EEF trial took place in a school setting, in which learners were 
supervised by teachers or teaching assistants while using GraphoGame 
Rime. If schools are going to use the programme for remote learning, it is 
important to consider how support will be put in place for learners. 


The full evaluation report for ‘GraphoGame Rime’ can be found here. 


Meta-analyses and systematic reviews included in the rapid evidence assessment 


Full list of included reviews 


Title Author Year Intervention type Age Pooled effect Outcome 
Digital game-based L2 learning outcomes for primary through 7 i : 
high-school students: A systematic literature review Acquah Emmanuel O; Katz Heidi T; 2020 Education games School aged N/A N/A 
The effects of ABRACADABRA on reading outcomes: An ne : ; ' 
: : : Ree Abrami Philip ; Borokhovski Eugene ; Computer assisted , Phonics 
ep ts analysis and landscape review of applied field Lysenko Larysa ; 2019 iaetiiciori Ages 3-9 0.263 (adjusted) 
Academic 
A Meta-Analytic Study Concerning the Effect of Computer- Batdi Veli - 2015 Computer assisted All (from school to 1.310 success in 
Based Teaching on Academic Success in Turkey : instruction adult) : secondary 
school 
A Bayesian Network Meta-Analysis to Synthesize the : SEM 
: PP Belland Brian R; Walker Andrew E; Kim Computer assisted All (from school to outcomes for 
Influence of Contexts of Scaffolding Use on Cognitive ; 2017 z : 0.740 
: : Nam Ju; instruction adult) elementary 
Outcomes in STEM Education school 
Synthesizing Results From Empirical Research on Computer- | Belland Brian R; Walker Andrew E; Kim 2017 Computer assisted All (from school to 0.460 STEM 
Based Scaffolding in STEM Education: A Meta-Analysis Nam Ju; Lefler Mason ; instruction adult) ‘ outcomes 
Bernard Robert M; Abrami Philip C; 
A Meta-Analysis of Three Types of Interaction Treatments in Borokhovski Eugene ; Wade C Anne; ; All (from school to : 
Distance Education Tamim Rana M; Surkes Michael A; 2008 Remote leaming adult) Discs AAI} 
Bethel Edward Clement; 
Are contextual and designed student-student interaction BOLOKOveM EUGENe > TaN uin ane; All (from school to 
icsaimientscadall oie indistanes-oducation? Bernard Robert M; Abrami Philip C; 2012 Remote learning adult) 0.310 Attainment 
aman ; Sokolovskaya Anna ; 
Boyle Elizabeth A; Hainey Thomas ; 
An update to the systematic literature review of empirical Connolly Thomas M; Gray Grant ; Earp All (from school to 
evidence of the impacts and outcomes of computer games Jeffrey ; Ott Michela ; Lim Theodore ; 2016 Education games adult) N/A N/A 
and serious games Ninaus Manuel ; Ribeiro Claudia ; Pereira 
Joao ; 
Digital game-based learning for K-12 mathematics education: Byun JaeHwan ; Joung Eunmi ; 2018 Education games School aged 0.370 Unclear 
A meta-analysis 
Learning 
A meta-analysis of the impact of technology on learning . : , outcomes in 
effectiveness of elementary students Chauhan Sumedha ; 2017 Remote learning Primary school 0.700 nontourial 
environments 
The Role of Collaboration, Computer Use, Learning ‘ : 
: - ete ; h Chen Juanjuan ; Wang Minhong ; Computer supported All (from school to 
pe ae and Supporting Strategies in CSCL: A Meta Kirschner Paul A; Tsai Chin-Chung ; 2018 collaborative learning adult) 0.420 Knowledge 
A meta-analysis examining the moderating effects of Chen Juanjuan ; Wang Minhong ; 2019 Computer supported All (from school to N/A N/A 
educational level and subject area on CSCL effectiveness Kirschner Paul A; Tsai Chin-Chung ; collaborative learning adult) 
The effectiveness of digital game-based vocabulary learning: Chen Meng-Hua ; Tseng Wen-Ta ; Hsiao 2018 Education games All (from school to 1.027 Vocabulary 
A framework-based view of meta-analysis Tsung-Yuan ; adult) 
The effectiveness of educational technology applications for Coraputerassisied Standardised 
enhancing mathematics achievement in K-12 classrooms: A Cheung Alan C K; Slavin Robert E; 2013 p School aged 0.160 tests in 


meta-analysis 


instruction 


mathematics 
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Computer-Assisted Second Language Vocabulary Instruction: 


Computer assisted 


All (from school to 


A Meta-Analysis COMO ma instruction adult) Da wesabulaly 
A Meta-analysis of the Peer Evaluation Effects on Learning : : st, : All (from school to Learning 
Achievements in Blended Learning Environment CulRanpatiZ feng: Eangin: eue pieneed eal ng adult) Cee achievement 
Effectiveness of online discussion strategies: A meta-analysis eee Aunteen Hang anya : 2013 Remote learning Puree noobie 0.499 Peariliig 
uryavanshi Rinki ; Yurekli Hulya ; adult) outcomes 
: : 0.39 (elaborated 
Effects of Feedback in a Computer-Based Learning Pe . . ‘ : 
Environment on Students’ Learning Outcomes: A Meta- Shi : Boat dle Hs oe Remco 2015 Computer assisted All (from aes to nee ae Learning 
Analysis ; Eggen Theo J : instruction adult) nowledge o outcomes 
correct results) 
B : . s Fang Ying ; Ren Zhihong ; Hu Xiangen ; Computer assisted Secondary and post- Maths learning 

A meta-analysis of the effectiveness of ALEKS on learning Giaesser Arthur C: 2019 fh etrietion secondary 0.100 euesnies 

. ‘ : Gerard Libby ; Matuk Camillia ; Computer assisted Learning 
Automated, adaptive guidance for K-12 education McElhaney Kevin ; Linn Marcia C; 2015 inatriction School aged 0.340 Gutcomes 
Serious games as new educational tools: how effective are , ; . : é 
they? A meta-analysis of recent studies Girard C ; Ecalle J ; Magnan A ; 2013 Education games Unclear N/A N/A 

-- ; : Hainey Thomas ; Connolly Thomas M; 
A systematic literature review of games-based learning : ae ie ’ ‘ 
empirical evidence in primary education Boyle Elizabeth A; Wilson Amanda ; 2016 Education games Primary school N/A N/A 
Razak Aisya ; 

2 ' Hussein Mahmood H; Ow Stew Hock; 

ae cocoa ee On Eementaly Cheong Loh Sau; Thong Meow-Keong ; 2019 Education games Primary school N/A N/A 

9: y Ebrahim Nader Ale; 

Ten years of Computer-Supported Collaborative Learning: A Jeong Heisawn ; Hmelo-Silver Cindy E; 2019 Computer supported All (from school to 0.490 STEM 

meta-analysis of CSCL in STEM education during 2005-2014 Jo Kihyun ; collaborative learning adult) : outcomes 

1:1 online tuition: a review of the literature from a pedagogical : ’ i All (from school to 

perspective Jopling M ; 2012 Remote learning adult) N/A N/A 

Updated Meta-Analysis of Learner Control Within Educational | Karich Abbey C; Burns Matthew kK; Maki Computer assisted All (from school to paiaitiesot 

apa 2014 . : 0.050 academic and 

Technology Kathrin E; instruction adult) 5 P 
ehaviour 

Effectiveness of Computer-Based Scaffolding in the Context : ; : 

: : pees : Kim Nam Ju; Belland Brian R; Walker Computer assisted All (from school to STEM 
of Problem Based Learning for Stem Education: Bayesian Andrew E: 2018 fictaetion adult) 0.385 Suiconies 
Meta-analysis 
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic f , . Computer assisted All (from school to Learning 
Review Kulik James A; Fletcher J D; 2016 istniction adult) 0.620 eiitcornes 
The Effects of Computer-Assisted Instruction in Reading: A I Computer assisted Reading 
Meta-Analysis mUNIKeL AMY"; euts instruction School aged pay outcomes 
Establishing an empirical link between computer-mediated Computer supported All (irom school to Language 
eae (CMC) and SLA: A meta-analysis of the Lin Huifen ; 2014 collaborative learning adult) 0.441 learning 
Intelligent Tutoring Systems and Learning Outcomes: A Meta- | Ma Wenting ; Adesope Olusola O; Nesbit Computer assisted All (from school to Learning 

: ha tengo 2014 F : 0.410 
Analysis John C; Liu Qing ; instruction adult) outcomes 
Digital games and the US National Research Council's Martinez-Garza Mario ; Clark Douglas B; 2013 Education games All (from school to N/A N/A 
science proficiency goals Nelson Brian C; adult) 
The Effectiveness of Online and Blended Learning: A Meta- Means Barbara ; Toyama Yukie ; Murphy 2013 Remote learning/Blended All (from school to 0.200 Learning 
Analysis of the Empirical Literature Robert ; Baki Marianne ; learning adult) : outcomes 
; 2 - ‘ ‘ F , Means Barbara ; Toyama Yukie ; Murphy Remote learning/Blended All (from school to Learning 
Evaluation of Evidence-Based Practices in Online Learning Robert : Bakia Marianne ; Jones Karla : 2009 learning adult) 0.240 Guiconies 
Effects of Computer-Assisted Instruction on the Mathematics : . F : F 
Performance of Students with Learning Disabilities: A SA eee Blane anys 2019 at eet aetaaas School aged N/A N/A 
Synthesis of the Research y } 
A Spotlight on Lack of Evidence Supporting the Integration of Poirier Mark ; Law Jeremy M; Veispak 2019 Blended learning School aged N/A N/A 


Blended Learning in K-12 Education: A Systematic Review 


Anneli ; 
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Challenges in the online component of blended learning: A 


Rasheed Rasheed Abubakar; Kamsin 


systematic review Amirrudin ; Abdullah Nor Aniza; 2020 Blended learning Unclear NA MA 

: ; : . Rigney Alexander M; Hixson Michael D; Computer assisted 
Headsprout: A Systematic Review of the Evidence Breven'Dariel/0; 2020 aetrletion School aged N/A N/A 

, ; F : Riopel Martin ; Nenciovici Lucian ; Potvin 

Impact of serious games on science learning achievement af : f ; ‘ 

: . : haha ; Patrice ; Chastenay Pierre ; Charland . All (from school to Declarative 
Sublette conventional instruction: an overview and Patrick : Sarrasin Jeremie Blanchette: 2019 Education games adult) 0.340 knowledge 

y Masson Steve ; 
A meta-analysis of the impact of computer technology versus 
traditional instruction on students in kindergarten through : 0.26 (distance : 
twelfth grade in the United States: A comparison of academic | Sandy-Hanson Anika E; 2006 Computer aesisted School aged education effect Beagemie 

‘ : wile : ae : instruction , outcomes 

achievement, higher -order thinking skills, motivation, physical size taken) 
outcomes and social skills 
Retrospect and prospect of computer assisted English oad ees di 2018 Computer assisted All (from school to 0.500 L2 learning 
language learning: a meta-analysis of the empirical literature Bahai: 9 ; instruction adult) } outcomes 
Effective reading programs for middle and high schools: A Slavin Robert E; Cheung Alan ; Groff 2008 Computer assisted Middle and high 0.100 fe 
best-evidence synthesis Cynthia ; Lake Cynthia ; instruction school : CAI 
A Meta-Analysis of the Effectiveness of Intelligent Tutoring : dene ie Computer assisted Maths learning 
Systems on K-12 Students' Mathematical Learning Steenbergen-Hu Salying ; Cooper Harris ; | 2013 instruction School aged aia outcomes 
A Systematic Meta-Analytic Review of Evidence for the : ; 

: " : F Strong Gemma k; Torgerson Carole J; Computer assisted % Reading 
sen as of the "Fast ForWord" Language Intervention Torgerson David : Hulme Charles ; 2011 ‘aetrlétion School aged 0.260 (pooled) 
Supporting Secondary School Students’ Reading ter Beek ; Marlies ; Brummer Leonie ; Crnpiitardesisiod 
Comprehension in Computer Environments: A Systematic Donker Anouk S; Opdenakker Marie- 2018 'p : Secondary school N/A N/A 

: saps ; instruction 

Review Christine J L; 
The differential effects of interactive versus didactic pedagogy | Thomas Tieja ; Alexander Kristopher ; 2013 Computer assisted All (from school to 0.175 Learning 
using computer-assisted instruction Jackson Renee ; Abrami Philip C; instruction adult) ‘ outcomes 
Effects of game-based learning on students’ mathematics Tokac Umit ; Novak Elena ; Thompson . Maths learning 
achievement: A meta-analysis Christopher G; euie Education games School aged Ot outcomes 
Digital game-based second-language vocabulary learning and SMA uae: ates . : All (from school to 
conditions of research designs: A meta-analysis study Tsai Yu-Ling ; Tsai Chin-Chung ; 2018 Education games adult) 0.986 Vocabulary 
Online Writing Instruction for Children with Disabilities: A : . Bete! ; 
Review of the Empirical Literature Vasquez Ill ; Eleazar ; Straub Carrie ; 2016 Remote learning School aged N/A N/A 
Learning Mathematics in Metacognitively Oriented ICT-Based | Verschaffel Lieven ; Depaepe Fien ; Computer assisted 
Learning Environments: A Systematic Review of the Literature | Mevarech Zemira ; ra instruction menodl-aged me nn 
Where is the evidence? A meta-analysis on the role of Domain 
argumentation for the acquisition of domain-specific Wecker Christof ; Fischer Frank ; 2014 pel saatsa aan d Unclear 0.000 specific 
knowledge in computer-supported collaborative learning g knowledge 
Effectiveness of Cognitive Skills-Based Computer-Assisted Weng Pei-Lin ; Maeda Yukiko ; Bouck Computer assisted Cognitive 

: ea tose vane é Pee 2014 : : School aged 0.350 learning 
Instruction for Students With Disabilities A Synthesis Emily C; instruction eiiteomies 
Earobics[R]. What Works Clearinghouse Intervention Report What Works Clearinghouse 2009 eer Primary school 0.490 Alphabetics 
Read Naturally. What Works Clearinghouse Intervention . Computer assisted F 
Report What Works Clearinghouse 2007 iG Primary school N/a N/a 
Technology and writing: Review of research Williams Cheri ; Beam Sandra ; 2019 pu scrote School aged N/A N/A 

; - ae Wouters Pieter ; van Nimwegen ; 

A Meta-Analysis of the Cognitive and Motivational Effects of Christof : van Oostendorp ; Herre ; van 2013 Education games All (from school to 0.290 Learning 


Serious Games 


der Spek ; Erik D ; 


adult) 


20 


A meta-analytic review of the role of instructional support in 


All (from school to 


Wouters Pieter ; van Oostendorp ; Herre . 

game-based learning ; 2013 Education games adult) 0.330 Knowledge 

: F , : Xu Zhihong ; Wijekumar Kausalai Kay; ; . 
The effectiveness of intelligent tutoring systems on K-12 ; : : i F Computer assisted Reading 
students’ reading comprehension: A meta-analysis Ramirez Gilbert ; Hu Xueyan ; Irey Robin | 2019 instruction School aged p.600 comprehension 
What makes the difference? A practical analysis of research eae tifs ee . : All (from school to Learner 
on the effectiveness of distance education Zhao Y; Lei J; YanB;LaiG; Tan HS; a momole learning adult) Og outcomes 
Digital'dame: based vocabulary Jeaming Wwuere.ateave and Zou Di ; Huang Yan ; Xie Haoran ; 2019 Education games Unclear N/A N/A 


where are we going? 


Note: where available the outcomes use the description provided by the authors. Terms such as “academic attainment 


” or “learner outcomes” may be equivalent. 
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Conclusion 


Key findings and implications 
1. Teaching quality is more important than how lessons are delivered 
Pupils can learn through remote teaching. 
Ensuring the elements of effective teaching are present — for example clear explanations, scaffolding and feedback — 
is more important than how or when they are provided. There was no clear difference between teaching in real time 


(“synchronous teaching”) and alternatives (“asynchronous teaching’). 


For example, teachers might explain a new idea live or in a pre-recorded video. But what matters most is whether the 
explanation builds clearly on pupils’ prior learning or how pupils’ understanding is subsequently assessed. 


2. Ensuring access to technology is key, particularly for disadvantaged pupils 
Almost all remote learning uses digital technology, typically requiring access to both computers and the internet. 


Many reviews identify lack of technology as a barrier to successful remote instruction. It is important that support is 
provided to ensure that disadvantaged pupils — who are more likely to face these barriers — have access to technology. 


In addition to providing access to technology, ensuring that teachers and pupils are provided with support and guidance 
to use specific platforms is essential, particularly if new forms of technology are being implemented. 


3. Peer interactions can provide motivation and improve learning outcomes 


Multiple reviews highlight the importance of peer interaction during remote learning, as a way to motivate pupils and 
improve outcomes. 


Across the studies reviewed, a range of strategies to support peer interaction were explored, including peer marking 
and feedback, sharing models of good work, and opportunities for live discussions of content. 


The value of collaborative approaches was emphasised in many reviews, although notably many studies involved older 
learners. Different approaches to peer interaction are likely to be better suited to different age groups. 


4. Supporting pupils to work independently can improve learning outcomes 


Pupils learning at home will often need to work independently. Multiple reviews identify the value of strategies that help 
pupils work independently with success. 


For example, prompting pupils to reflect on their work or to consider the strategies they will use if they get stuck have 
been highlighted as valuable. 


Wider evidence related to metacognition and self-regulation suggests that disadvantaged pupils are likely to particularly 
benefit from explicit support to help them work independently, for example, by providing checklists or daily plans. 


5. Different approaches to remote learning suit different tasks and types of content 


Approaches to remote learning vary widely and have different strengths and weaknesses. Teachers should be 
supported to consider which approaches are best suited to the content they are teaching and the age of their pupils. 


For example, games for learning were found to have a high impact on vocabulary learning in foreign languages, but 
there is less evidence related to their use in other subjects. 


Likewise, using technology to support retrieval practice and self-quizzing can help pupils retain key ideas and 
knowledge, but is not a replacement for other forms of assessment. 
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The evidence in this review is drawn from diverse contexts that do not closely parallel the circumstances 
facing schools responding to Covid-19 in 2020. Many forms of digital technology could in theory be used to 
support remote learning, but are typically used in schools and have not been evaluated as remote learning 


tools. In all cases, it is important for teachers and school leaders to use their professional judgement in 
determining the support they provide their pupils and to monitor its impact on learning. 


Research questions 
The systematic search retrieved 60 systematic reviews and meta-analyses. 
1. How effective are distance learning and online learning approaches in comparison to usual schooling? 


There is evidence that remote learning can have positive impacts on learner outcomes — most meta-analyses conducted 
on remote learning approaches found results to be either higher or the same as traditional instruction. The evidence, 
however, is spread across a wide range of different interventions, subjects and study designs. 


This review does not attempt to measure the impact of distance learning overall, but instead summarises the impact of 
different reviews and approaches thematically. 


2. Does pupil or school level disadvantage moderate pupil achievement outcomes in distance learning 
approaches? 


Very few studies examined the impact of remote learning approaches on disadvantaged pupils. More research is 
needed to see if remote learning approaches widen the attainment gap. Several studies do identify access to technology 
as an important barrier to implementing remote learning approaches. If school or pupil level disadvantage means that 
pupils do not have access to the technology needed to engage with remote learning, this is likely to cause attainment 
gaps to widen. 


3. How do different distance learning approaches moderate pupil achievement outcomes? 


The methodological limitations of this review of existing systematic reviews and meta-analyses means that it is not 
possible to systematically examine differential impacts of different approaches. There are several trends within the 
reviews in which research teams have highlighted particularly promising approaches, or conducted moderator analysis 
that indicate differential effects. 


- Across multiple approaches and reviews, aspects of collaborative learning or peer to peer interaction are 
identified as leading to improved pupil outcomes. 

- Scaffolding, feedback and opportunities to reflect on learning are highlighted as promising aspects of remote 
learning, across a variety of approaches. 

- Computer assisted instruction and games for learning have some evidence of promise when examined as 
general approaches but are lacking in high quality evidence when applied to remote learning. 


4. What are the characteristics of effective distance learning implementation? 


Studies that examined specific differences in distance learning implementation found limited results. Evidence was 
mixed and inconclusive between synchronous, asynchronous or mixed instruction. There was no evidence that any one 
medium or delivery mechanism was particularly successful at improving pupil outcomes. There was evidence that 
quality of implementation did impact learning outcomes and that this might include providing professional support for 
practitioners delivering remote instruction. 


5. Which EEF-funded programmes that could be or have been delivered remotely have shown promise in 
terms of improving pupil achievement, compared to business as usual? 


No EEF-funded programmes have been delivered in genuine remote learning conditions. There are, however, several 
projects that might be possible to deliver remotely, with adaptation. For example, projects like ABRA have the capability 
to be delivered remotely due to supporting independent work and delivering content through the internet. However, in 
the EEF trial of ABRA, teaching assistants played an active role in supporting the learners. 


The results of less promising programmes may also not be directly transferable to remote learning, as when evaluated 
by the EEF the comparison condition was often high quality usual practice in schools. For example, GraphoGame Rime 
was not found to have an impact on early reading outcomes, but the comparison group were pupils identified with low 
levels of reading, who were receiving extensive support through other interventions such as small group tuition. 
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In times of school closure, programmes that had less promising results may lead to positive outcomes in comparison 
with alternatives in remote provision. This might particularly be the case for disadvantaged pupils or pupils with low prior 
attainment who may have been receiving additional support in normal school conditions. 


Limitations 
This review has been limited by a number of factors. 
Umbrella review methodology 


In order to complete this review quickly and to respond to schools need for evidence on remote learning, this review 
was limited to existing systematic reviews and meta-analyses. This means that we are unable to perform moderator 
analysis on specific types of remote learning and cannot apply quality criteria to individual studies. The limitations of 
this methodology mean that we have not provided an overall estimate of impact for remote learning, or for any of the 
specific approaches through which distance learning may be delivered. 


Limitations in the evidence-base on distance learning 


Many of the reviews failed to find evidence that related to school-aged learners receiving remote teaching. Many reviews 
that included studies from schools also included studies on higher education and these were usually greater in number. 
In these combined reviews, it was often unclear in moderator analysis how many studies from school-aged pupils 
featured in the analysis. Other approaches, such as computer assisted instruction or learning games had much more 
school-aged evidence, but evaluations rarely took place during periods of remote learning. Careful consideration about 
the extent to which the evidence from higher education remote learning studies and school-aged non-remote learning 
studies can be applied to the current circumstances. 


Quality of review methods within education 


One of the interesting findings of the review was the low quality of review methods amongst the retrieved studies. We 
rated the quality of each of the reviews using a number of criteria (the full rating system is available in the protocol 
here). Almost none of the studies applied any kind of risk of bias assessment for the included studies. This meant that 
45 of the 60 reviews were rated as “low” in the quality assessment. While some reviews did explore methodological 
causes for heterogeneity — for example, sample size or type of measure — the fact that almost no studies assessed the 
quality of the individual studies included in the reviews is a severe limitation to the quality of the reviews summarised in 
this rapid evidence assessment. 


Future research 


A technical write up of the rapid evidence assessment will be published in the future. 
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Additional resources 


The Education Endowment Foundation has created a number of resources that are relevant to supporting learners 
during the Covid-19 outbreak. All resources can be found here. Some of the resources that directly relate to the findings 
of the remote learning rapid evidence assessment are detailed below: 


guidance report 


Resource Description Link 
Guidance EEF guidance reports provide clear and actionable | htips://educationendowmentfoundation.org.uk/tools/guidance- 
reports recommendations for teachers on a range of high- reports/ 

priority issues based on the best available 

evidence. 
Parental Four recommendations on working with https://educationendowmentfoundation.org.uk/tools/guidance- 
engagement parents to support their child’s learning. reports/working-with-parents-to-support-childrens-learning/ 


Parental 


The underlying evidence review for the 


https://educationendowmentfoundation.org.uk/evidence- 


evidence review 


engagement parental engagement guidance report. summaries/evidence-reviews/parental-engagement/ 

evidence review 

Digital Four recommendations on using digital https://educationendowmentfoundation.org.uk/tools/guidance- 
technology technology to improve children’s learning. reports/using-digital-technology-to-improve-learning/ 
guidance report 

Digital The underlying evidence review for the digital | https://educationendowmentfoundation.org.uk/evidence- 
technology technology guidance report. summaries/evidence-reviews/digital-technology-2019/ 


Metacognition 
guidance report 


Seven recommendations for teaching self- 
regulated learning and metacognition, 


https://educationendowmentfoundation.org.uk/tools/guidance- 
reports/metacognition-and-self-regulated-learning/ 


Teaching and 
Learning 
Toolkit 


The Teaching and Learning Toolkit provides 
an accessible summary of the evidence 
across 35 different approaches aimed at 
improving pupil outcomes 


https://educationendowmentfoundation.org.uk/evidence- 
summaries/teaching-learning-toolkit/ 


Peer tutoring 


Toolkit summary of peer tutoring approaches 


https://educationendowmentfoundation.org.uk/evidence- 
summaries/teaching-learning-toolkit/peer-tutoring/ 


Metacognition 


Toolkit summary of metacognition and self- 


https://educationendowmentfoundation.org.uk/evidence- 


Clearinghouse 
list of studies 


remote learning approaches, identified by the 
What Works Clearinghouse 


and self- regulation approaches summaries/teaching-learning-toolkit/meta-cognition-and-self- 
regulation regulation/ 
Parental Toolkit summary of parental engagement https://educationendowmentfoundation.org.uk/evidence- 
engagement approaches summaries/teaching-learning-toolkit/parental-engagement/ 
Homework Toolkit summary of homework (primary and https://educationendowmentfoundation.org.uk/evidence- 
secondary) summaries/teaching-learning-toolkit/homework-primary/ 
https://educationendowmentfoundation.org.uk/evidence- 
summaries/teaching-learning-toolkit/homework-secondary/ 
Digital Toolkit summary of digital technology https://educationendowmentfoundation.org.uk/evidence- 
technology approaches summaries/teaching-learning-toolkit/digital-technology/ 
EEF funded This is the full list of evaluations that have https://educationendowmentfoundation.org.uk/projects-and- 
evaluations been funded by the EEF. evaluation/projects/ 
What Works A list of studies that examine the impact of https://ies.ed.gov/ncee/wwe/distancelearningstudy 
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